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Abstract 

The blocking probability of a network is a common measure of its 
performance. There exist means of quickly calculating the block- 
ing probabilities of Banyan networks; however, because Banyan 
networks have no redundant paths, they are not inherently fault- 
tolerant, and so their use in large-scale multiprocessors is prob- 
lematic. Unfortunately, the addition of multiple paths between 
message sources and sinks in a network complicates the calcula- 
tion of blocking probabilities. A methodology for exact calcula- 
tion of blocking probabilities for small networks with redundant 
paths is presented here, with some discussion of its potential use 
in approximating blocking probabilities for large networks with 
redundant paths. 
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1 Introduction 

The M.I.T. Artificial Intelligence Laboratory's Transit Project is develop- 
ing high-performance networks for large-scale parallel computers. One of 
the project's aims is to make these networks fault-tolerant. This has been 
difficult in traditional multistage interconnection networks for parallel com- 
puters, because these networks have often been Banyan networks [9, 10]. 
Banyan networks do not have redundant paths [2], and thus the failure of 
a switching element will necessarily cut off communication between at least 
one message source and one message sink in the network. 

The addition of redundant paths can enhance fault-tolerance. Unfortu- 
nately, it also creates problems for the traffic theorist. In a Banyan net- 
work, if one assumes messages at the inputs are generated by independent 
processes, the presence or absence of messages at the inputs of any switch 
in the network is independent of the presence or absence of messages at 
the other inputs of that switch. Thus the analysis of blocking probabilities 
in Banyan networks is simplified [5]. When redundant paths are allowed, 
independence is violated. 

A similar problem has been studied in the context of telephone switching 
systems [4]. However, in telephone switching systems the model is one of 
a circuit-switched network where the holding time for circuits varies. Fur- 
thermore, in the methods described in [4], it is assumed that the networks 
modeled are symmetric; because there are classes of asymmetric networks 
that are of interest, and because we are partly interested in calculating 
blocking probabilities in the presence of (asymmetric) faults, these methods 
are not satisfactory. 

The number of equations that must be solved in order to find the ex- 
act blocking probability of a general network is probably exponential in the 
number of communications channels entering a stage in a network; however, 
there appears to be no proof of this in the literature. Still, even if large prob- 
lems are intractable, a method of exact solution is useful because it allows 
some evaluation of approximation methods through comparison with exact 
solutions for small problems, and also because it can be used as the basis 
for a Monte Carlo approximation method. This is the approach taken by 
Harvey and Hills in [3]. Harvey and Hills were considering circuit- switched 
telephone networks with unique paths; but their approach, which was to 
find approximate solutions of exact equations, rather than exact solutions 
to approximate equations, can still be of use here. 








Figure 1: An 8x8 deterministically-interwired network with redundant 
paths. There are a number of different paths from any source to any sink, 
to increase fault- tolerance; redundant paths from message source 4 to sink 
3 are highlighted. Routing is oblivious, with stochastic concentration. This 
wiring scheme is from [1]. 



2 Problem Statement 

Consider a multistage, unbuffered, synchronous, packet-switched network 
with redundant paths. Such a network might look like the one depicted 
in Figure 1. Messages enter the network on the channels leading from the 
message sources, which in Figure 1 axe depicted on the left side and labeled 
Iq through If. The network has multiple stages: if we consider the stage 
consisting of all the sources to be stage 0, then stage 1 consists of the column 
of switching elements connected directly to the sources; stage 2 the column 
of switches to the right of stage 1, etc. The message sinks are the nodes on 
the right side, labeled Oq through O7. 

The processes generating messages at the sources are independent and 
memory less. With some specified probability p,-, each source i generates 
or fails to generate a single message at the beginning of each cycle. Each 
generated message is directed to a stage 1 switch via one of the channels 
connecting the source to the network. The channel by which to send the 
message is chosen randomly, and each channel has the same probability of 



being chosen; i.e., routing is oblivious. The network is synchronous: at each 
cycle messages move from stage i to stage z + 1. It is also unbuffered: if a 
message is blocked at some stage, it is considered to be lost, and does not 
in any way affect the future states of the system. 

In the networks we model, the set of output channels of a switching 
element is divided into nonempty disjoint subsets called logical directions. 
At each cycle, the switching element directs each incoming message in one 
logical direction. We assume uniform addressing here; messages are equally 
likely to be addressed in any one of the logical directions. We use stochastic 
concentration'. 

• If there are fewer messages or exactly the same number of messages 
directed in the logical direction as there are channels in that logical 
direction, then the channels that will carry the messages are chosen 
randomly, with uniform probability. 

• If there are more messages directed in a logical direction than there 
are channels in that direction, the messages that can be carried are 
chosen with uniform probability, and the other messages are blocked 
and lost. 

We refer to a switching component with M input channels and N logical 
directions, each consisting oi K channels, as an u MxN, dilation K switch." 1 

Having denned our model, let us return to the network of Figure 1. The 
switches here are 4x2, dilation 2 switches, except at the last stage, where 
they are simply 2x2 (dilation 1) switches. In the 4x2, dilation 2 switches, 
the top two output channels constitute one logical direction, and the bottom 
two constitute another. 

We may state our question as follows: What is the probability that an 
arbitrary message entering the network at a channel leading from a source 
reaches a channel leading to a sink? We answer this question by finding the 
probability mass functions of the loads on channels leading to sinks. 



x One such switch is the RNl switching component, designed at the M.I.T. Artificial 
Intelligence Laboratory's Transit Group. The RNl switching component, intended for use 
in interconnection networks for parallel multiprocessors, is an 8 x 4, dilation 2 switch. It 
is described further in [7]. 
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Figure 2: Interstage wiring. Note that no subset of the channels depicted 
need be mutually independent in a network with redundant paths. 

3 The Joint Probability Mass Function of an Ag- 
gregate of Channels 

Suppose that the input channels of a switch 5, depicted in Figure 2, are 
connected to several switches Ri, jR 2 , • • • , Ri- Let us use the random variable 
L to denote the entire output loading configuration of S at some specified 
discrete time t, so that ¥{L = 1} is the probability that the output channels 
of the switch have some particular loads designated in their aggregate by / 
during cycle t. 

Now consider the loads on the input channels Cn, . . . , C{ w at cycle t-1. 
(Because we assume a synchronous, unbuffered network with memory less 
processes generating the messages at the inputs, only the cycle before cycle 
t is of interest.) Let us denote the loads on the input channels at cycle t-1 
with the random variables Lc lx , • . . , Lc iw ■ 

In order to find the joint probability mass function of the loads on the 
output channels of 5, we condition on the loads on the input channels: 

P{£ = /}= Y, P{J = '|£c„=fc„,...,Jc<. = !<*.}• 

P-f^Cu = hu , • • • , L Ciw = l Ciw } (l) 

where the sum is over all tuples l Cll , . . . , h iw with elements in {0, 1}. 




Figure 3: Although the probabilities of the message loads on the channels 
Cn, . . . , Ci w are not in general independent, the loads on the subset of chan- 
nels from each switching element are independent given the message loads 
on the input channels Bn, . . . , J5^„ 



Suppose that we can compute ¥{L = I \ L Cll = h n , • • • ,L Cxw = h iw }- 2 
In order to compute the probability of an output loading configuration of 5 
we will still need to find the joint probability mass function of the channel 
loads Leu , • • • , Lc iv> ■ In a Banyan network, it would be easy to compute this 
function; it would simply be the product of the probability mass functions of 
the loads on the individual channels, as channel loads in a Banyan network 
are independent. 3 In a network with redundant paths, however, the loads 
on these channels are not in general independent, as they may derive from 
the same sources, and a message from a single source that has traveled one 
path in the network cannot be traveling along another path. Thus another 
method must be used. 

In Figure 3, we see that the input channels Cu,.. .,C{ W of switch S 
are the output channels of switches Ri,...,R{. Let us call the loads on 
the input channels to these switches Lb xx , . . . , Lb u • We may now calcu- 
late P{£c n - kiii-'-iLCiv = h iw ] by conditioning on the values of the 
variables Lb^ , . . . , Lb u • We have 

P{X Cll = lc lli --- i Lc iw = h iw } = 



An expression for this conditional probability is derived in Appendix A. 
3 See Appendix B. 



]T P{X Cll - /c„ , . . • , ic iw = h tw I X-Bn = /s n , • • • , L Bit = hu } • 

p {^n ='b 11 »---»^, = 'b.-,} (2) 

where the sum is over all tuples l Bll , • . ■ , l Bit with elements in {0,1}. 

The loads on the output channels of these switches are not in general mu- 
tually independent. However, let us partition them into subsets according to 
the switch at which they originate, so that for the channels shown in Figure 3 
we would have the subsets {<7 n , . . . C lu } , {<7 2 i, • • • C 2v ] , • • • , {Cii, . . -,C iw }. 
Note that, under the assumption of uniform addressing, and given the loads 
on the channels 5n, . . . ,Ba, the loads on the switch output channel sub- 
sets are mutually independent. That is, if we know the input loads for the 
switches -Ri,...,i2i, then the loading probabilities for the output channels 
of each of the switches do not depend on the output loads of any other 
switch. We may use this fact to derive the joint probability mass function of 
the loads on the output channels Cn, . . . ,d w by conditioning on the input 
channel loads. We have then 

P{£c„ = l Cll i-..,L Ciw = k iw } = 

zl F \ L Cii = Id i»---, L Clu =i Clu I L Bll = l Bll ,-..,L Blr = l Blr j ■ 

P{£c 21 = k 21 ,---,Lc 2v =i C2v \ Lb 21 = h 21 ,...,L B2i = l B2t j • 

P{X Cll = hn , • . • , L Ciw = lc iw I L Bil = l Bil , . . . , L Btt = l B . t } . 
P{X flll =l Bll ,...,L Bit =l Bit } (3) 

where the sum is once again over all tuples l Bll ,.. . ,/jg. t with elements in 
{0,1}. 

The conditional probabilities can be evaluated as described in Appendix A, 
and ~P{L Bll = l Bll , . . . ,L Bit = l Bit } can be evaluated recursively by means 
of Equation (3), until the channels Bu,...,Bi t correspond to sources. If 
these channels originate at message sources, then we substitute instead the 
probability mass functions corresponding to sources. We may simply take 
the product of these functions for the sources in question, as in our model 
the processes generating messages at the sources are mutually independent. 

If source i, depicted in Figure 4, generates a message with probability 
Pi and has k channels into the network, then we have for the loads on the 
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Figure 4: The source I{ generates a single message at each cycle with prob- 
ability pi. The message is transmitted with uniform probability over a ran- 
domly picked channel in the set {Ci, . . . , Ck}- 



channels C\, . . . , Ck the joint probability mass function 
Y{L Cl =--l Cl ,...,L Ck =l Ck }= < 



1 - Pi if all the /c . are 
Y if exactly one I a, is 1, 

and the rest are 



(4) 



otherwise 



4 Automatic Calculation of Blocking Probabili- 
ties 

It will be clear that the automatic calculation of blocking probabilities by 
this means will require a great deal of time. Suppose we have a computer 
program that calculates the blocking probabilities for a network in the most 
obvious way, by finding the joint probability mass function of the channels 
leaving the final stage, using Equation (3) recursively. In the worst case, we 
can imagine a network where there are N stages and M dependent channels 
between each of the JV stages, and the joint probability mass function of all 
of the channels between each of the stages must be formed. The domain 
of the joint probability mass function for each stage then is of size 2 M , 
each value being calculated as a sum over 2 M terms. Assuming the time to 
calculate each of the terms summed over in Equation (3) is 0(M), we have 
then OfiVM2 2M J for the worst-case performance. 

The performance on some networks can be better than this, however. 
Suppose that we need to calculate V{Lc x - lc x > • • • ^c n = h n }- Let S (c) 
denote the set of source nodes from which messages can reach channel c. 
If we can partition the set of channels {Ci,...,C n } into disjoint subsets 



5i, . . . , 5 m such that for any d G S { and C 2 E 5,-, t 7^ j, S (d) nS(C 2 ) is 
empty, then the loads on the channels in each subset Si are independent of 
the loads on the channels in any and all of the other subsets in the partition. 4 
Then the expression ¥{Lc x = hn-- ■ ->L Cn = h n } can be factored into the 
product of m joint probability mass functions, one for each subset Si. In the 
limiting case of a Banyan network, a complete factoring will be possible for 
every set of channels, and the summation itself can be factored, so that the 
worst case performance for a Banyan network of N stages with M channels 
between the stages becomes O(NM). 

A program has been written to evaluate the joint probability mass func- 
tion of the loads on specified channels in a multistage interconnection net- 
work. The program is given a symbolic description of the interconnection 
network. The program uses the network representation to build an internal 
structure in which (for example) information about independence of chan- 
nel loads has been precomputed, and channels have been assigned names 
generated from the names of the their nodes of origin and destination. One 
can then assign message generation probabilities to the sources and query 
the program for the probability mass function of interest. The result is 
numerical, as in the example below: 

> (setq d8x8 (parse-multistage-network 

deterministically-interwired-8x8-rep)) 
#<MULTISTAGE-HETWORK 8x8> 

> (jlpmf '(tt6-o7-0 tt7-o7-0) d8x8) 
(#S(JLPHF-PART CHAMHELS (#<CHAHHEL TT6-07-0> 

#<CHAHIEL TT7-07-0>) 

HUMBER-OF-CHAIHELS 2 

VECTOR #(10321939817/17179869184 
2931771091/17179869184 
2931771091/17179869184 
994387185/17179869184) ) ) 

Here we have calculated the joint probability mass function of the loads on 
two channels leading from two 2x2 switches to sink 01 in the network 
of Figure 1, given a probability of transmission in each message source of 
1/2; we assume here, as in [l], that a message sink can receive two messages 
during a single cycle. 

To find the blocking probability of the network, we can form the proba- 
bility of successful message transmission as the ratio of the expected number 

4 As will be seen from the argument in Appendix B. 



of messages entering the network to the expected number of messages ar- 
riving at sinks. Because of the symmetry of the network, all the channels 
leading to sinks have identical loading probabilities, and so we can simply 
sum the expectations of their loads. We have then that the expected number 
of messages arriving at a single sink is 

1 • P{XtT6-07 = l 5 £TT7-07 = 0} + 1 • P{£tT6-07 = 0,2/TT7-O7 = 1} 

+ 2 • P{£tT6-07 = l?£TT7-07 = 1} 

« 0.457 

and the expected number of messages arriving at all sinks during any cycle 
is then 8 • 0.457 « 3.66. 

Because the expected number of messages entering the network is 8 • | = 
4, we have that the aggregate probability of successful message transmission 
in this network at a loading factor of 1/2 is 

E[messages arriving at sinks] 



E[messages injected by sources] 

and thus the blocking probability is approximately 0.086. 

We plot for the network of Figure 1 the probability of successful message 
transmission versus the probability that a source transmits in Figure 5. 

The implementation internally records joint probability mass functions 
so that they need not be recomputed. Although the asymptotic performance 
is in general pessimal, the implementation has been coded with some atten- 
tion to performance, because much of the same code is likely to be used in 
an approximation scheme. 

5 Conclusions; Future Work 

We have presented a means of exact calculation of the blocking probability 
of a multistage network with redundant paths, and demonstrated its use in 
a program that automatically calculates blocking probabilities and exploits 
independence of channel loading probabilities where this is possible. 

The implementation described cannot be used to calculate the blocking 
probabilities of networks with much more path redundancy than the one 
of Figure 1. We might consider an implementation that could exploit the 
symmetry exhibited by some multistage networks, but such an implemen- 
tation could still not be used on a network like that in Figure 6, in which 
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Figure 5: The probability of successful message transmission (P{Success}) 
plotted against the the source transmission probability (Pi) for the network 
of Figure 1. 

the wiring in the first and second stages is not symmetric and is in fact 
randomly generated. That such networks are of interest is demonstrated in 
[6]. 

We are investigating the possibility of approximately solving these sys- 
tems of equations. Our hope is that the approximate solution of the equa- 
tions will yield approximations of higher confidence in fewer steps than would 
direct simulation of the network. 
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Figure 6: A 16 X 16 network with random interwiring in the first and second 
stages. The figure is from [1]. 
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A Probability Mass Functions of Switching Com- 
ponent Output Channels 

Suppose we have an M X iV, dilation K switch. We may form the conditional 
probability mass function of the loads on the output channels, given the 
input load, by conditioning. Say that the random variable L^ g represents 
the load on the g th channel in the / th logical direction. Now for ease of 
representation, let us say that the random variable L represents the entire 
output configuration of the switch, and takes on values / that are N X K 
arrays whose elements lf, g are if the channel in question carries no message, 
or 1 if it carries a message. Thus in our notation the event we denote by 

{£ = <} 

is equivalent to the event 

{ -£i,i = fi,i» -£1,2 = h,2i • • • » Li f K = hjCi 

-^2,1 = h,l)L2,2 = h,2,- — >L2,K = h,K> 

Ln,i = fo,i» Ln,2 = fjv,2> • • • j -kjv.fc = In,k } 

We wish to evaluate the expression P{L = / | Lc x = lei > • • • » Lc M = lc M }- 
The switches that we are modeling do not distinguish in any way between 
messages arriving at distinct input channels. Thus the loads on the individ- 
ual input channels Ci, . . . , Cm are not individually of significance, but their 
sum is. Say that ]£i=i /c, = J- For simplicity of expression, while working 
in an event space where Lq x + . . . + Lc M = J, we use the superscript j. 
Thus we say P{£ = I | L Cl = lc x , • • • , L Cm = h M } = ?{L j = I}. Now, to 
form the conditional probability P{£ = / | J = j}, we condition on the num- 
ber of messages directed in each logical direction. If the random variable 
Ci represents the number of messages routed in logical direction i, we can 
condition: 

?{L j = l}= £ J>{L> = l\D 1 = d 1 ,...,D N =d n }. 

P{D 1 = d 1 ,...,D N = d n } (5) 

where the sum is over all iV-tuples di,...,d N such that each d{ > and 
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Under uniform addressing, the probability that of the j arriving mes- 
sages, d\ are directed in direction 1, cfo in direction 2, and so on, is simply 
multinomial, so that 

Now let us evaluate V{L 3 = I | D\ = d\, . . . , Dn = d n }. Say that c; is the 
number of messages output in direction i when V — /; that is, C{ — Y^=i h,g- 
This number is not the same as d;, because if there are more than K messages 
to be output in a iiT-wide direction, some messages are dropped and lost. 
If C{ messages are output, then under stochastic concentration the channels 
are picked with uniform probability, and so the probability of any single 
configuration will be l/(^). Thus 



\V = l\D 1 = d 1 ,...,D N = d N } = 



if for any i,c t - ^ min (d;, K) 

N 1 (7) 

TT -j?- otherwise ^ ' 



where c t = J2g=i kg- 

Combining Equations (5), (6), and (7), we have 

P{L = I | L Cl = l Cl ,.--,Lc M = lc M } = 

' N 



where c; = Y^=i h, g , j = Y^iiih^ and the sum is over the iV-tuples 
di, . . . ,djv such that Y,iLi <k = j and for each i, c; = min (d;, K). 

B Loads on Banyan Network Channels at a Sin- 
gle Stage are Independent 

A proof for the special case of delta networks is presented in [8]; here we 
present a different proof for the general case. 

The proof is entirely straightforward. Note first that, if messages are 
generated at source nodes by mutually independent random processes, and 
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the sets of messages on distinct channels entering a switching node origi- 
nate at disjoint sets of source nodes, then the loads on those channels are 
necessarily independent. 

We now claim that the sets of messages on distinct channels entering 
any switching node in a Banyan network satisfy this criterion: i.e., they 
originate at disjoint sets of sources. 

For, consider: if channel A and channel B are two channels entering a 
switching node, and a message on channel A and a message on channel B 
originate at a single source, then it must be the case that at least two paths 
exist from that source to any sinks accessible from the switching node: one 
path that uses channel A and one that uses channel B. But this is impossible 
in a Banyan network, as Banyan networks are in fact those in which there 
is exactly one path from each source to each sink. 

Thus the sets of messages on distinct channels entering any switching 
node in a Banyan network must originate at disjoint sets of sources, and so 
the loads on the channels entering any switching node in a Banyan network 
must be mutually independent, as was to be proved. 
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